Turning a Mobile Device into a Mouse in the Air

A mouse has been one of the most successful user interfaces due to its intuitive use. As more devices are equipped with displays and offer rich options for users to choose from, a traditional mouse that requires a surface to operate is no longer sufficient. While different types of air mice are available in the market, they rely on accelerometers and gyroscopes, which significantly limit the accuracy and ease of use. In this paper, we develop a system that can accurately track hand movement to realize a mouse. A unique advantage of our scheme is that it achieves high tracking accuracy using the existing hardware already available in the mobile devices (e.g., smart phones and smart watches) and equipment to be controlled (e.g., smart TVs). More specifically, our approach sends inaudible sound pulses at a few selected frequencies, and uses the frequency shifts to estimate the speed and distance traveled. We then develop techniques to quickly calibrate the distance between speakers and narrow down the device’s initial position using its movement trajectory. Based on the information, we continuously track the device’s new position in real time. This is feasible because many devices, such as smart TVs, PCs, and laptops, already have multiple speakers. When only one speaker is available, we can leverage the frequency shift of sound along with the phase of received WiFi signal to enable tracking. Our evaluation and user study demonstrate that our system achieves high tracking accuracy (e.g., median error of around 1.4 cm) and ease of use.